Skip to content

feat: Add control plane telemetry - scheduler eligible workers histogram - #682

Open
Angela (Angelawork) wants to merge 5 commits into
agent-substrate:mainfrom
Angelawork:feature/eligible-workers-telemetry
Open

feat: Add control plane telemetry - scheduler eligible workers histogram#682
Angela (Angelawork) wants to merge 5 commits into
agent-substrate:mainfrom
Angelawork:feature/eligible-workers-telemetry

Conversation

@Angelawork

Copy link
Copy Markdown
Collaborator

Fixes #564 (Part 3)

  • Tests pass
  • Appropriate changes to documentation are included in the PR

Description

This PR implements Part 3 of #564 by adding telemetry histogram instrumentation for ate.scheduler.eligible_workers in ateapi. It measures unassigned free worker capacity remaining after all scheduling constraint filters are applied, sampled at every scheduling decision.

Key Changes:

  • Updated Scheduler.Schedule() to record eligible candidate workers per pool (recordEligibleWorkers).
    Defined SchedulingConstraintKey (ate.scheduling.constraint) and constraint classification values (none, required_nodes, selector) in internal/ateattr/ateattr.go.
  • Added unit tests in scheduling_test.go covering candidate counts, namespaced attributes, zero-capacity fleet states, empty fleets, sandbox class mismatches, draining workers, and constraint classifications.

Testing

  • go test -buildvcs=false ./cmd/ateapi/internal/controlapi/...
  • go test -buildvcs=false ./cmd/atenet/internal/router/...
  • make test

E2E Test

  • ./hack/create-kind-cluster.sh
  • ./hack/install-ate-kind.sh --deploy-ate-system --deploy-demo-counter
  • ./hack/run-e2e.sh ./internal/e2e/suites/metrics/...

@Angelawork
Angela (Angelawork) force-pushed the feature/eligible-workers-telemetry branch from 5b29a8f to 1ff8de8 Compare August 4, 2026 15:52
@Angelawork
Angela (Angelawork) marked this pull request as draft August 4, 2026 17:51
@JeffLuoo

Copy link
Copy Markdown
Collaborator

Please rebase and fix e2e tests

@Angelawork
Angela (Angelawork) force-pushed the feature/eligible-workers-telemetry branch 2 times, most recently from 2cc4a44 to 59434d7 Compare August 4, 2026 18:23
@Angelawork
Angela (Angelawork) marked this pull request as ready for review August 4, 2026 18:23
@Angelawork
Angela (Angelawork) force-pushed the feature/eligible-workers-telemetry branch from ad9eac4 to 74dce74 Compare August 4, 2026 18:27
@Angelawork
Angela (Angelawork) force-pushed the feature/eligible-workers-telemetry branch from 74dce74 to 8f82992 Compare August 4, 2026 18:29
Comment thread internal/e2e/suites/metrics/metrics_test.go Outdated
Comment thread cmd/ateapi/internal/scheduling/scheduling.go Outdated
Comment thread cmd/ateapi/internal/scheduling/scheduling.go Outdated
Comment thread cmd/ateapi/internal/scheduling/scheduling.go Outdated
Comment thread cmd/ateapi/internal/scheduling/scheduling.go Outdated
Comment thread cmd/ateapi/internal/scheduling/scheduling.go Outdated
@Angelawork
Angela (Angelawork) force-pushed the feature/eligible-workers-telemetry branch from e86f58b to abf9345 Compare August 5, 2026 21:13

// RegisterEligibleWorkers registers the ate.scheduler.eligible_workers histogram instrument against meter.
// Returns (nil, nil) if meter is nil. If registration fails, an error is returned.
func RegisterEligibleWorkers(meter metric.Meter) (metric.Int64Histogram, error) {

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You don't need to make RegisterEligibleWorkers a public function because it's only used in this file.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also consider using the name newEligibleWorkers because the term Register usually infer a callback.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Feature request: More system metrics for debuggability (Activation SLI, crash accounting, and capacity signals)

2 participants